{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Platform's Data-Object API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The platform’s Simple-Object API enables performing simple data-object operations that resembles Amazon’s Simple Storage Service (S3) API. In addition to the S3-like capabilities, the Simple-Object Web API enables appending data to existing objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: The Python API for accessing the data objects is a tech preview and is included in the current release as a sneak peek to future release features but without official support in this release. Note that tech preview features don’t go through QA cycles and might result in unexpected behavior. Please consult the Iguazio support team before using these features.\n",
    "As an alternative, you can access the [data-object operations REST API](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/simple-object-web-api/data-object-operations/) which offers equivalent functionality and is officially supported."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import v3io.dataplane"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataplane client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "v3io_client = v3io.dataplane.Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: You can pass to the client the `endpoint` and `access_key` parameters explicitly. The following code is equivalent to the default values:\n",
    ">\n",
    ">``` python\n",
    ">from os import getenv\n",
    ">v3io_client = v3io.dataplane.Client(endpoint='http://v3io-webapi:8081',\n",
    ">                                    access_key=getenv('V3IO_ACCESS_KEY'))\n",
    ">```\n",
    ">\n",
    "> When calling externally, you can obtain the URL of your cluster by copying the API URL of the web-APIs service (webapi) from the **Services** dashboard page. You can select between two types of URLs:\n",
    "- **HTTPS Direct** (recommended) &mdash; a URL of the format `https://<tenant IP>:<web-APIs port>`; for example, `https://default-tenant.app.mycluster.iguazio.com:8443`.\n",
    "- **HTTPS** &mdash; a URL of the format `https://webapi.<tenant IP>`; for example, `https://webapi.default-tenant.app.mycluster.iguazio.com`.\n",
    ">\n",
    "> For more information see the [Data-Service Web-API General Structure](https://www.iguazio.com/docs/latest-release/reference/api-reference/web-apis/data-service-web-api-gen-struct/) documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Number of maximum parallel connections**: Another noteworthy parameter is `max_connections`, which defines the number of maximum parallel connections when performing batch operations. If left unspecified, the default is 8 connections. For more information see the [Put Multiple Objects](#Put-Multiple-Objects) section in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All data in the platform is stored in user-defined data containers. In this case we use the predefined \"users\" container. For more information refer to [Data containers, collections, and objects documentation](https://www.iguazio.com/docs/latest-release/concepts/containers-collections-objects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONTAINER = 'users'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data path where to store the kv table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import getenv, path\n",
    "\n",
    "V3IO_USERNAME = getenv('V3IO_USERNAME')\n",
    "OBJECTS_PATH = path.join(V3IO_USERNAME, 'examples', 'v3io', 'objects')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Put Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adds a new object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"It was the best of times,\\n\\\n",
    "it was the worst of times,\\n\\\n",
    "it was the age of wisdom,\\n\\\n",
    "it was the age of foolishness,\\n\\\n",
    "it was the epoch of belief,\\n\\\n",
    "it was the epoch of incredulity,\\n\\\n",
    "\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to iguazio/examples/v3io/objects/The Period.txt\n",
      "Status code: 200\n"
     ]
    }
   ],
   "source": [
    "OBJECT = path.join(OBJECTS_PATH, 'The Period.txt')\n",
    "print(f'Writing to {OBJECT}')\n",
    "response = v3io_client.object.put(container=CONTAINER, path=OBJECT, body=text)\n",
    "print(f'Status code: {response.status_code}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieves an object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It was the best of times,\n",
      "it was the worst of times,\n",
      "it was the age of wisdom,\n",
      "it was the age of foolishness,\n",
      "it was the epoch of belief,\n",
      "it was the epoch of incredulity,\n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = v3io_client.object.get(container=CONTAINER, path=OBJECT)\n",
    "print(response.body.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Append"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `put` method supports appending data to an existing object.\n",
    "\n",
    "> **Note**: The option to append data is extension to the S3 PUT Object capabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "text2=\"it was the season of Light,\\n\\\n",
    "it was the season of Darkness,\\n\\\n",
    "it was the spring of hope,\\n\\\n",
    "it was the winter of despair,\\n\\\n",
    "\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status code: 200\n"
     ]
    }
   ],
   "source": [
    "response = v3io_client.object.put(container=CONTAINER, path=OBJECT, body=text2, append=True)\n",
    "print(f'Status code: {response.status_code}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It was the best of times,\n",
      "it was the worst of times,\n",
      "it was the age of wisdom,\n",
      "it was the age of foolishness,\n",
      "it was the epoch of belief,\n",
      "it was the epoch of incredulity,\n",
      "it was the season of Light,\n",
      "it was the season of Darkness,\n",
      "it was the spring of hope,\n",
      "it was the winter of despair,\n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = v3io_client.object.get(container=CONTAINER, path=OBJECT)\n",
    "print(response.body.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete Object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deletes an object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n"
     ]
    }
   ],
   "source": [
    "response = v3io_client.object.delete(container=CONTAINER, path=OBJECT)\n",
    "print(response.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Put Multiple Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the highest possible throughput, we can send many requests towards the data layer and wait for all the responses to arrive (rather than send each request and wait for the response). The SDK supports this through batching. Any API call can be made through the client's built in `batch` object. The API call receives the exact same arguments it would normally receive (except for `raise_for_status`), and does not block until the response arrives. To wait for all pending responses, call `wait()` on the `batch` object:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note**: The number of parallel connections is determined by the `max_connections` parameter when you created the client. For instance, to set 16 parallel connections you should have in the beginning of the notebook `v3io_client = v3io.dataplane.Client(max_connections=16)`. The default is 8 connections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Template of word sequence\n",
    "\n",
    "nouns = ['time', 'person', 'year', 'way', 'day', 'thing', 'man', 'world', 'life', 'hand', 'part', 'child', 'eye', 'woman', 'place', 'work', 'week', 'case', 'point', 'government', 'company', 'number', 'group', 'problem', 'fact']\n",
    "adjectives = ['good', 'new', 'first', 'last', 'long', 'great', 'little', 'own', 'other', 'old', 'right', 'big', 'high', 'different', 'small', 'large', 'next', 'early', 'young', 'important', 'few', 'public', 'bad', 'same', 'able']\n",
    "prepositions = ['to', 'of', 'in', 'for', 'on', 'with', 'at', 'by', 'from', 'up', 'about', 'into', 'over', 'after']\n",
    "others = ['the', 'that', 'this', 'my', 'one']\n",
    "\n",
    "sequence = [nouns, prepositions, others, adjectives, nouns]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "company of the same life\n",
      "world for that same way\n",
      "number into one first point\n",
      "woman to the first man\n",
      "world from one good case\n",
      "man into one different world\n",
      "place up this good fact\n",
      "thing into my right life\n",
      "day for this last year\n",
      "eye of this big government\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "random.seed(42)\n",
    "\n",
    "#Generate sequence of words\n",
    "\n",
    "for i in range(10):\n",
    "    generated_text = \" \".join([random.choice(values) for values in sequence])\n",
    "    print(generated_text)\n",
    "    v3io_client.batch.object.put(container=CONTAINER, path=path.join(OBJECTS_PATH, f'obj_{i:02}'), body=generated_text)\n",
    "\n",
    "# wait for all writes to complete\n",
    "responses = v3io_client.batch.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The looped `put` interface above will send all `put` requests to the data layer in parallel. When `wait` is called, it will block until either all responses arrive (in which case it will return a `Responses` object, containing the `responses` of each call) or an error occurs - in which case an exception is thrown. You can pass `raise_for_status` to `wait`, and it behaves as explained above.\n",
    "\n",
    "> Note: The `batch` object is stateful, so you can only create one batch at a time. However, you can create multiple parallel batches yourself through the client's `create_batch()` interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the contents of the first object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "company of the same life\n"
     ]
    }
   ],
   "source": [
    "response = v3io_client.object.get(container=CONTAINER, path=path.join(OBJECTS_PATH, 'obj_00'))\n",
    "print(response.body.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the file system to list the new objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "obj_* files in /v3io/users/iguazio/examples/v3io/objects:\n",
      "obj_00\n",
      "obj_01\n",
      "obj_02\n",
      "obj_03\n",
      "obj_04\n",
      "obj_05\n",
      "obj_06\n",
      "obj_07\n",
      "obj_08\n",
      "obj_09\n"
     ]
    }
   ],
   "source": [
    "from os import sep\n",
    "import pathlib\n",
    "\n",
    "V3IO_OBJECTS_PATH = path.join(sep, 'v3io', CONTAINER, OBJECTS_PATH)\n",
    "\n",
    "print(f\"obj_* files in {V3IO_OBJECTS_PATH}:\")\n",
    "for file in pathlib.Path(V3IO_OBJECTS_PATH).glob(\"obj_*\"):\n",
    "    print(file.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Delete the Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the file-system interface to delete a objects directory from the relevant data container:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "shutil.rmtree(V3IO_OBJECTS_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively you can use the following commands:\n",
    "```\n",
    "!rm -r $V3IO_OBJECTS_PATH\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
